This section details on the original issue you should resolve

<issue_title>WVA Limitations of Cluster Wide Usability</issue_title>
<issue_description># Summary

When installing WVA - cluster wide - as in the only supported mode of installation currently - a user will be limited to using only one llm-d stack. Consequently, if a user decided to install more than one llm-d stack (in a separate namespace from the initial llm-d stack) with hopes of leveraging WVA, they will notice their existing scaled model variants disappear because the helm installation will have overridden those resources.

A proposed solution is to provide two separate, distinct types of installation:

• An installation path to install just the WVA controller
• An installation path to install just the model specific resources - in a just-in-time/as needed fashion

I propose another solution, but it is a bit more opinionated; I will elaborate on the solutions in the sections below.

Detailed Description of the Problem

Lets start off with what works

The below architecture resembles a cluster wide installation of one instance of the WVA controller that can monitor and scale variants of a single model in a single namespaces in a single llm-d stack. This works perfectly well with the current helm installation, in fact, routinely well, thank you @clubanderson:

• WVA Namespace

  • wva-controller

• Model Namespace A:

  • llm-d stack
    • Model A
  • wva-resources
    • va
    • hpa
    • vllm-service
    • servicemonitor

Now lets see what does not work

The below architecture resembles a cluster wide installation of one instance of the WVA controller. Assume that I am now adding a new model to a new, seperate, llm-d stack in a separate namespace, and I rerun the automation that exists today for WVA, you will then see that the below desired scenario is not possible:

• WVA Namespace

  • wva-controller

• Model Namespace A:

  • llm-d stack
    • Model A
  • wva-resources
    • va
    • hpa
    • vllm-service
    • servicemonitor

• Model Namespace B:

  • llm-d stack
    • Model B
  • wva-resources
    • va
    • hpa
    • vllm-service
    • servicemonitor

So here is what really happens. When having an existing model variant being scaled, in this case Model-A in namespace A, the helm installation to create the variant for Model B in the namespace B will remove all of the wva resources in namespace A. Thereby leaving the user with the following new architecture:

• WVA Namespace

  • wva-controller

• Model Namespace A:

  • llm-d stack
    • Model A

• Model Namespace B:

  • llm-d stack
    • Model B
  • wva-resources
    • va
    • hpa
    • vllm-service
    • servicemonitor

The problem here is the tight coupling of controller and model variant resources in the same helm installation. The consequence is that the current v0.4.2 WVA installation can really only support a single model in a single llm-d stack in a single namespace, as shown above. So let's view the solutions.

Detailed Proposed Solution

Solution A

Decouple the WVA controller installation from the WVA model variant resource installation. In other words:

• An installation path to install just the WVA controller
• An installation path to install just the model specific resources - in a just-in-time/as needed fashion

Allow a user to just and only install then WVA controller - in effort to prime the cluster to get ready to deploy model variants on an as needed basis.

Then allow the user to install the model variant resources for a particular model when they are needed. Sometimes they are needed immediately at the time of the WVA Controller other times they are needed after the fact. The crux of the matter is - the controller and model variant resources are separate items and should be respected as such during the installation.

Solution B

Pick an installation path a stick to it. Currently - there are talks of allowing multiple WVA controllers per cluster by separating them via namespace, i.e., the following:

• Model Namespace A:
  • wva-controller
  • llm-d stack
    • Model B
  • wva-resources
    • va
    • hpa
    • vllm-service
    • servicemonitor

The above architecture should actually in complete theory solve the limitation of the the cluster wide installation since EACH helm installation of WVA and it's model specific resources is scoped to a single llm-d stack and model.

If there is going to be an offering for a cluster wide installation support will be needed for the items claimed in this issue.

If there is no real need or benefit for a cluster wide installation why should it be maintained or pro...